Papers by Toby Jia-Jun Li
Interactive Task Learning from GUI-Grounded Natural Language Instructions and Demonstrations (2020.acl-demos)
Copied to clipboard
| Challenge: | SUGILITE is an intelligent task automation agent that can learn new tasks and relevant associated concepts interactively from the user’s natural language instructions and demonstrations using GUIs. |
| Approach: | They propose to use third-party mobile apps to teach new tasks and concepts using verbal instructions and demonstrations. |
| Outcome: | The proposed system can learn new tasks and relevant concepts from user's natural language instructions and demonstrations, and it generalizes taught concepts to different contexts and task domains. |
LLMs for Now, Fine-Tuning for Later: An Ensemble Approach to Data Drift in Domain-Specific Tasks (2026.acl-srw)
Copied to clipboard
Yuxuan Lu, Bingsheng Yao, Shao Zhang, Yisi Sang, Yun Wang, Hansu Gu, Peng Zhang, Tun Lu, Toby Jia-Jun Li, Dakuo Wang
| Challenge: | Deploying machine learning models in domain-specific scenarios is challenged by data drift and the scarcity of expert annotations. |
| Approach: | They propose a system that combines an LLM, an AL-assisted compact model and an automatic switch module to assist the active learning process. |
| Outcome: | The proposed system achieves 96–98% switch accuracy and outperforms both models used alone. |
Leveraging Variation Theory in Counterfactual Data Augmentation for Optimized Active Learning (2025.findings-acl)
Copied to clipboard
| Challenge: | Active Learning (AL) allows users to provide focused annotations to integrate human preferences and domain knowledge into machine learning models. |
| Approach: | They propose a counterfactual data augmentation approach inspired by Variation Theory to generate targeted variations along key conceptual dimensions. |
| Outcome: | The proposed approach achieves significantly higher performance when there are fewer annotated data, showing it can address the cold start problem in Active Learning. |
OPeRA: A Dataset of Observation, Persona, Rationale, and Action for Evaluating LLMs on Human Online Shopping Behavior Simulation (2026.acl-long)
Copied to clipboard
Ziyi Wang, Yuxuan Lu, Wenbo Li, Amirali Amini, Bo Sun, Yakov Bart, Weimin Lyu, Jiri Gesi, Tian Wang, Jing Huang, Yu Su, Upol Ehsan, Malihe Alikhani, Toby Jia-Jun Li, Lydia Chilton, Dakuo Wang
| Challenge: | evaluating LLMs' ability to mimic real user behavior remains an open challenge due to the lack of high-quality, publicly available datasets that capture both the observable actions and the internal reasoning of an actual user. |
| Approach: | They propose a dataset of Observation, Persona, Rationale, and Action collected from real human participants during online shopping sessions. |
| Outcome: | The proposed dataset is the first to evaluate how well current LLMs can accurately simulate the next web action of a specific user. |
EyeMulator: Improving Code Language Models by Mimicking Human Visual Attention (2026.acl-long)
Copied to clipboard
Yifan Zhang, Chen Huang, Yueke Zhang, Jiahao Zhang, Toby Jia-Jun Li, Collin McMillan, Kevin Leach, Yu Huang
| Challenge: | Code Language Models learn attention based on statistical input-output token correlations. |
| Approach: | They propose a model-agnostic technique to align CodeLLM attention with human visual attention without architectural changes. |
| Outcome: | The proposed model outperforms baselines in three languages, with gains of over 30 CodeBLEU points in translation and up to 22 BERTScore points in summarization. |
Towards a Design Guideline for RPA Evaluation: A Survey of Large Language Model-Based Role-Playing Agents (2025.findings-acl)
Copied to clipboard
| Challenge: | Role-Playing Agents (RPAs) are increasingly popular due to diverse task requirements and agent designs. |
| Approach: | They propose an evidence-based evaluation design guideline for LLM-based RPAs based on agent attributes, task attributes, and evaluation metrics. |
| Outcome: | The proposed evaluation design guideline is based on a systematic review of 1,676 papers published between Jan. 2021 and Dec. 2024. |